feat: RFC-0045 step 7b, enable reference and repeater fields - #223

RFC-0045 rollout step 7b: enables reference and repeater fields in production, wherever the renderer serving a site reads them. See RFC §9 step 7 and the new "Amended in 7b" section.

Production call sites and the capability decision

MCP (update_collection, change_field_type inputs) and Studio (/api/instance features.referenceFields) already key on store.fieldTypes !== undefined (steps 5 and 6b), so they now list the types on the admin Worker, Node and the playground. The store still decides per site.

One-time entry_refs rebuild job (§2.4)

• rebuildEntryRefs (core-ts/src/references/rebuild-job.ts, exported from @cloudhal/core-ts). Rows come from entryRefItemsOf, split out of the import rebuild (references/rebuild.ts), so the job uses rebuildRows with fromKey, oldShapePairs and the text↔reference rules, and the holder's source_status.
• Deviation: a chunk of up to 100 entries uses one DELETE and chunked INSERTs (each below 90 KB). Each row is joined to nodes on (id, read hash) through json_each items. This anchors each node exactly as one statement pair per node would, but in a fixed number of statements per chunk, which fits the free tick (40 statements). It never deletes site-wide.
• Every statement carries the site anchor: the site node's hash, read with the chunk (round 1; the collections text could reach 256 KB per statement). A definitions change, or any other site edit, fails the whole chunk, which is then re-read.
• The job leaves a site alone while a text↔reference conversion runs there (operation_running, cursor unmoved; round 1), because the rows decide what phase B accepted.
• New table entry_refs_state (site_id, cursor, done, updated_at) (migration 0034_entry_refs_state, added to SITE_TABLES).
  • The cursor is nodes.key. The advance is an upsert guarded on the cursor as read.
  • The last chunk removes orphans (source_id NOT IN (SELECT id FROM nodes WHERE site_id = ?)) and marks the site done.
• Wiring, like lists.backfill:
  • the admin Worker's refs.rebuild system job (cloudflare/src/entry-refs-job.ts): every 5 min at :02:30, 12 statements on Free and 150 on Paid, two reads once done;
  • openLocalStore at start-up.
  • lists.backfill has no CLI command or route, so this job has neither, and no API route was added.
• Accepted: on Workers Free, while a site still has work, the job can push another job due in the same tick to the next minute. comments.test.ts now starts from the steady state.

Gate tests and the step-1 watch note

• testing-gates-fields.ts runs from gatesSuite on SQLite and D1. It covers import, pull, push (published: target) and restore (restoreBundle).
  • Each is refused without a capability, with one that says no, and with one for another site only.
  • Each is accepted with the site's capability, and rebuilds rows.
• Watch note: heldSiteBodies (step 1/2) already reads the held site bodies that new snapshots name. A new test shows that a site without the capability cannot point a new snapshot at a capable site's held body. It fails when heldSiteBodies is disabled.
• API: admin.test.ts push is accepted with the site's capability (200) and refused with another site's (409).
• workerd e2e (cli/test/cloudflare-gates.test.ts): a push with a reference field is accepted against the real site Worker (fields: 2) and refused with 409 fields_unsupported against the old one.
• Job suite testing-entry-refs-job.ts runs on SQLite, Miniflare D1 and DO SQLite. It covers:
  • resume by cursor;
  • the statement budget;
  • orphans, scoped to their own site;
  • R17: a writer racing the batch keeps its fresh rows and the moved node is skipped;
  • a definitions change between read and batch: the chunk writes nothing and is re-read;
  • with a running rename and a held old-shape value, the rows equal both the writer's and the import rebuild's (source_status included).
• cloudflare/test/entry-refs-job.test.ts checks, on D1, the declared cost per run, the logging, and the steady state.

Other changes

• CopySiteOptions.fieldTypes accepts undefined, so callers can pass local.store.fieldTypes.
• cli/test/fields-cli.test.ts: the local store now accepts a retype to a repeater (dry run).
• docs/guides/fields.md covers both types:
  • targets, several/max and conditions;
  • sub-fields and item labels;
  • limits;
  • the delete lock and "Used by";
  • operations: sub, conversions and "Clear and finish";
  • MCP: strict schemas, fieldsStrict/clearFields, translations and wholeLists;
  • EntryFields.astro rendering;
  • where the types are available, and the one-time check.
• Updated "until step 7" wording in ADR-0046, the astro-renderer README and code comments. FIELD_TYPES stays the nine types; Studio and MCP already add REFERENCE_FIELD_TYPES behind the capability.
• node scripts/mcp-tools-doc.ts now renders from defineTools({ referenceTypes: true }). The README table is unchanged because it shows first sentences only.
• Changeset rfc45-step7b-enable.md: patch (0.x rule). It says the types are now enabled.

Review round 1 (42a50ef)

1. The capability is per site Worker. A test covers two bindings, where SITE reports fields: 2 and SITE_1 does not: the site routed to SITE_1 is refused and the main site is accepted. Gates use the same per-binding rule.
2. The job waits while a text↔reference conversion runs. Test: rows that phase B accepted stay; the job then finishes once the operation is done. The test fails without the fix.
3. The site-hash anchor replaces the collections text. Test: with collections over 90 KB (about 130 KB) and an entry with 20 references, the job finishes within the Free budget (≤ 12 statements a run, every statement < 90 KB).
4. The guide now says a move is refused with reference_target_moved. I checked its other codes against api/src/errors.ts.

Tests run after round 1, one file at a time:

• the job suite on SQLite, D1 and DO (8/8 each);
• the whole Cloudflare package (15 files);
• cloudflare-gates (9/9) and cloudflare-multisite (6/6) on workerd;
• biome, size check and typecheck.

Local checks (first round)

• pnpm check (run 2): every step before the tests passed (biome, size check, font fallbacks, MCP tools doc, demo archive, typecheck). In the test stage at --concurrency=3, two Studio UI files failed under machine load; both pass alone. Turbo then stopped before core-ts, api, cli and cloudflare.

• turbo run test --concurrency=1 --continue: 17 of 19 packages passed, Studio included (85 files). The other two:

  • core-ts store.test.ts: only the D1 variants failed, with EADDRNOTAVAIL. One D1 pass fills this machine's port range (more than 8,000 sockets in TIME_WAIT), a known local issue.
  • cli studio-site-build.test.ts: an 11 s build timeout under load. It passes alone twice (7/7).

• core-ts store.test.ts run again in slices, waiting for the ports to drain between slices:

  • SQLite 286 tests;
  • D1: 231 in SiteGraphStore, 7 time-gate tests, 48 in the other suites;
  • DO: 231 in SiteGraphStore, plus 7.

  All pass, with no EADDRNOTAVAIL.

• cli/test/cloudflare-gates.test.ts on workerd: 9/9.

Not merged; waiting for the PM review.

🤖 Generated with Claude Code